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Introduction 



In recent years, growing numbers of educators have become interested in approaches to science 
teaching and learning involving projects or inquiry (e.g., Pea, 1993; Ruopp, Gal, Drayton, & 
Pfister, 1993) which harken back to Dewey and the Progressive era (Cremin, 1961). Such project- 
based approaches are touted as a means of promoting students' active engagement with science in 
ways that are recommended by situated (e.g.. Brown, Collins, & Duguid, 1989; Lave & Wenger, 
1991) and social constructivist views of learning (e.g., Newman, Griffin, & Cole, 1989; 

Vygotsky, 1978). Typically, these project-based approaches also stress the importance of learning 
the "process" of science, such as formulating empirically investigable questions and supporting 
claims with evidence. This is contrasted with other approaches which stress learning the "content" 
of science, such as conceptual understanding of established scientific theories. As a participant in 
the Learning through Collaborative Visualization (CoVis) project (Pea, 1993; Pea & Gomez, 1992) 
over the past few years, I have frequently encountered a criticism based on the opposition of the 
inquiry or project approach with a relative emphasis on process or "doing science" to approaches 
with a relative emphasis on understanding scientific concepts. The criticism can be paraphrased as 
follows: "Why are you spending all this effort to train 'little scientists', when so few of the 
children are going to end up becoming scientists? You'd be better off focusing on helping them 
gain a conceptual understanding of the various domains of science, especially since there is so little 
time for all the coverage necessary in the science curriculum." This criticism begs the fundamental, 
vedue-laden question; "Why do we want children to learn science?" Put within the context of 
education in a democracy, this question can be framed as, "What kind of scientific literacy is 
necessary or desirable for citizens in today's world?" 

In this paper, I will describe an argument in support of "training little scientists" based on the 
requirements of participation in a media-laden, democratic society fraught with scientific claims and 
counterclaims. This leads to a framework for evaluating the effectiveness of various approaches to 
science education at addressing the requirements specified, as well as their enactments in practice. 
Finally, I will demonstrate the promise of this framework by examining some results of case study 
and survey research in project-based science classes at the high school level. 

Scientific literacy for today's society 

In order to understand some important aspects of scientific literacy for today's society, I want to 
describe a scenario. Imagine you are in the midst of trying to conceive your first child. After two 
years with no conception, you have begun to contact various fertility clinics in your area. One 
Sunday, you open up the local newspaper and you find that there are concerns about a number of 



practices at fertility clinics, and uses of specific drugs for fertility purposes. Being personally 
invested in the issue, you read the newspaper article from start to finish. Some of the accounts are 
alarming. People have gotten seriously ill and even died unexpectedly. Health care providers have 
not always warned patients of known risks. But you, personally, are considering some of these 
drugs, and you desperately want a child. What are you to do? 

This scenario is not purely fictional to me, because I have two close friends to whom this actually 
happened. Let's step back from the scenario to the concerns of educators. How would we hope an 
educated adult who is not a biomedical research scientist would react to this article, which appeared 
in the Boston Globe (Kong, 1996)7 Perhaps a critical reading of the article would be helpful. The 
article contains a table tallying all deaths and adverse reactions to a number of fertility drugs 
reported to the FDA over a 25 year period. The author quotes one expert, who reacted to the table 
by saying "any reports of deaths made to the FDA need to be carefully interpreted" (p. A34). But 
the author is careful to point out that that particular expert represents a drug company which 
manufactures and distributes fertility drugs. This is a useful caution against biased sources, but the 
expert's caution seems a reasonable one. What would a careful interpretation of numbers of 
reported deaths and other "adverse reactions" entail? Part of a careful interpretation might include 
consideration of the other medical circumstances or conditions present in the 45 deaths and 1982 
total adverse reactions associated with Clomid in reports to the FDA between 1970 and 1995. 
Another consideration is which adverse reactions got reported to the FDA and which did not? How 
many adverse reactions normally occur during pregnancy? None of these things are possible to 
glean from the article or table, which is careful to mention "reports do not prove that the drugs 
caused all negative reactions" (p. A34). 

At another point, the author cites an account of a woman who was "perfectly healthy" until taking a 
drug, prompting her sister to form a "victims network." As if to bolster the claim that many others 
are experiencing problems with the drug, the sister reports that "thousands of people have 
contacted her organization" (Kong, 1996, p. A34). What kind of evidence is that? How many 
readers assume that the thousands of contacts translate into thousands of problems? How many 
question the relevance of the number of contacts? 

The newspaper account plants strong doubt in the scientific establishment's ability and willingness 
to fully test drugs' short and long-term consequences, implying that greed may be clouding the 
judgment of decision-makers in fertility clinics, research labs, and drug companies. Again, this 
caution seems warranted, but there is a flip side to the issue not mentioned in this article. Many 
citizens' rights groups, beginning with groups concerned with developing treatments for AIDS and 
HIV infection, have argued vehemently for "fast-tracking" important new drugs. Within such an 
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environment, it appears inevitable that individual citizens will be affected by matters that are 
marked by contradictory scientific studies such as the case of fertility drugs cited here. In addition 
to fertility and AIDS treatments, recent debates have raged around the danger of living near power 
lines, hormone treatment at menopause, and the optimal frequency of mammograms while citizens 
are in the midst of making decisions about those same issues. Even if the trend toward "fast- 
tracking" which citizens and corp>orations have been pushing were to end, philosophers of science 
from widely differing perspectives agree that scientific progress inevitably involves ongoing 
contentiousness and debate. For example. Popper (1959) argued for a strict logic of scientific 
theory building and testing which relied on the verifiability and refutability of theories; in his 
system, empirical investigations which refute theories will always be possible and should often be 
pursued, because knowledge develops through the debate that ensues. On the other hand, 
Feyerabend (1993) argues for a much less logical system of scientific method, which includes 
sometimes inconsistent argumentation, but is always characterized by competing theories and 
observations. 

Neil Postman (1992) claims that American society today can be characterized as a "Technopoly," in 
which individual citizens are expected to bow unquestioningly to the proclamations of experts from 
tiny domains that no one other than the highly trained can understand. He cites the tendency of 
Americans to believe any claim, no matter how outlandish, as long as it originated from a "study" 
done at a reputable research institution (p. 57). He recommends that citizens resist this trend, and 
do not believe in the "magical powers of numbers" nor "regard calculation as an adequate substitute 
for judgment" (p. 184). I would like to build on Postman's point and argue that citizens who do 
not understand how scientific research is done, and how scientific research is questioned on its 
own terms^, have very little recourse but to either (1) bow to the recommendations of their favorite 
expert or friend, and completely ignore the arguments grounded in scientific studies; or (2) follow 
the recommendations of the last scientific study they encountered. 

Although the techniques of Western science are surely not the only means of reaching adequate 
judgments about issues related to science, one means for citizens to participate in the conversations 
of experts is by gaining an understanding of research design and the statistical techniques used in 
studies grounded in this tradition. As occurs only too often in education, Dewey (1910/1964) made 
a similar point at the beginning of this century, and its importance has only increased: 



t Postman would not necessarily embrace my recommended curriculum, which differ from his own (see below in the 
section "The promise of project-oriented pe^gogy"). He argues for similar goals, however, when he says smdents 
should learn about induction, scientific theories and models, and conditions for validity (p. 193). 



"... science has been taught too much as an accumulation of ready-made material with 
which students are to be made familiar, not enough as a method of thinking, an attimde of 
mind, after the pattern of which mental habits are to be transformed." (p. 183) 



The "ready-made material" of current scientific concepts can only go so far, because science is 
constantly moving forward (resulting in an explosion since Dewey's time in the amount of material 
generally crammed into science curricula, which only exacerbates the problem). This point holds as 
much for physical science, environmental science, and social science as for the biomedical science 
I have focused on up to this point Adult citizens participating in a democratic society need to 
understand the dominant "methods of thinking" from Western science, or they will be unable to 
participate in the debates based in this tradition. We cannot expect all citizens to know the details of 
debates in these subjects as well as experts who have devoted years of study to each of these 
disciplines, but we can strive to enable citizens to at least become "legitimate peripheral 
participants" (Lave & Wenger, 1991) in the debates that affect their lives. 

A framework for teaching and learning today's scientific literacy 

As argued in the previous section, today's scientific literacy should enable students to participate in 
discussions about and critical evaluations of ongoing research results based in the traditions of 
Western science. In the recently completed National Science Education Standards (National 
Research Council (NRC), 1996), a similar point is made; science literacy is important because 
"Americans are confronted increasingly with questions in their lives that require scientific 
information and scientific ways of thinking for informed decision making" (p. 11, my emphasis). 
According to the Standards, one way students will develop scientific ways of thinking is by 
developing the "abilities necessary to do scientific inquiry", which include 

• Identify questions and concepts that guide scientific investigations. 

• Design and conduct scientific investigations. 

• Use technology and mathematics to improve investigations and communications. 

• Formulate and revise scientific explanations and models using logic and evidence. 

• Recognize and analyze alternative explanations and models. 

• Communicate and defend a scientific argument. (Content Standard A, Grades 9-12, pp. 
175-176) 

In order to "design and conduct scientific investigations", as well as "use logic and evidence" in 
ways they are often used in science, I propose some more specific important abilities. Students 
should be able to; 

1 . formulate empirically investigable research questions and problems, including common 
"epistemic forms" (Collins & Ferguson, in press) of research such as testing against the 
null hypothesis, isolating variables, and building simulations or models 

2 . create and recognize explicit and strong links between empirical data and claims 

3 . recognize threats to the validity and reliability of claims 
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4. understand the implications of research claims, including generalizability and limits, as 
well as issues such as statistical significance vs. power (i.e., "twice the risk" means 
different things when the base rate is 1% and the base is 40%, although in both cases 
statistical significance has been reached) 

5. understand the role of debate, consensus and multiple studies taking varied approaches 
to research issues. 

These points should be useful both in the design of curriculum and pedagogy and the evaluation of 
enacted curriculum. I will now turn to such an evaluation of enacted curricula based on the goals 
and concerns laid out here. 

The promise of project-oriented classrooms 

As mentioned above, Neil Postman (1992) has identified many of the same problems I have which 
today's citizens face, spiecifically the flood of scientific studies being conducted and how ill- 
equipped citizens taught science as an "accumulation of ready-made material" are to understand or 
debate the studies' relevance. In order to address these problems. Postman recommends teaching 
more history of science and more philosophy of science in school. Another approach is to have 
students directly "take a hand in the making of knowledge" (Dewey, 1910/1965, p. 188) by 
conducting science inquiry in the form of research projects. 

The National Science Education Standards (NRC, 1996) emphasize "inquiry" as vital to the science 
curriculum. By "inquiry" they mean 

the diverse ways in which scientists study the natural world and propose explanations 
based on the evidence derived from their work. Inquiry also refers to the activities of 
students in which they develop knowledge and understanding of scientific ideas, as well as 
an understanding of how scientists study the natural world (p. 23). 

The emphasis on inquiry, however, "should not be interpreted as recommending a single approach 
to science teaching ... conducting hands-on science does not guarantee inquiry, nor is reading 
about science incompatible with inquiry" (NRC, 1996, p. 23). Nonetheless, the Standards 
describe "successful science classrooms" as places where 

teachers and students collaborate in the pursuit of ideas, and students quite often initiate 
new activities related to an inquiry. Students formulate questions and devise ways to 
answer them, they collect data and decide how to represent it, they organize data to generate 
knowledge, and they test the reliability of the knowledge they have generated. As they 
proceed, students explain and justify their work to themselves and to one another, learn to 
cope with problems such as the limitations of equipment, and react to challenges posed by 
the teacher and by classmates. Students assess the efficacy of their efforts— they evaluate 
the data they have collected, re-examining or collecting more if necessary, and making 
statements about the generalizability of findings. They plan and make presentations to the 
rest of the class about their work and accept and react to the constructive criticism of others. 
(NRC, 1996, p. 33) 



This description follows much of what we in CoVis describe as a "project-oriented" pedagogy; by 
this we mean an approach to science teaching and learning which depends to some degree on 
students conducting science research projects with the help of their teacher. The Standards and the 
CoVis project both subscribe to a form of Dewey's "learning by doing" philosophy. Dewey 
claimed that "only by taking a hand in the making of knowledge, by transferring guess and opinion 
into belief authorized by inquiry, does one ever get a knowledge of the method of knowing" for 
science (1910/1964, p. 188). In the analysis below, I will evaluate this claim, and seek to 
understand whether and how "taking a hand in the making of knowledge" through science research 
projects results in a "knowledge of the method of knowing" for science. 

Data sources and methods 

The science teaching and learning I will consider comes from two sources: case study research in a 
project-based classroom (Polman, 1997) and a survey instrument known as the "Project Planning 
Exercises" administered to classes varying in the degree of their "project orientation." 

The case study research focuses on open-ended research projects conducted over a twelve week 
period at the end of one and a half years of intensive participant observation by the author (the time 
period was November-January in the 1995-96 schoolyear). It is part of a larger interpretive case 
study (Polman, 1997) conducted from 1994 through 1996 in Rory Wagner's^ class, tis part of both 
of our involvement in the Learning through Collaborative Visualization (CoVis) project (Pea, 

1993). The term interpretive is based on Erickson (1986), and refers to any form of participant 
observational research that is centrally concerned with the role of meaning in social life, enacted in 
local situations. One of the central features of the class weis that students conduct Earth Science 
projects of their own design', what this meant in practice was that they participjated in the 
formulation of a research question, the gathering of data to provide empirical evidence for 
addressing the question, analysis of those data, and reporting in both written and oral format. Data 
collection techniques included written field notes and videotapes of classroom observation at each 
project phtise, collection of artifacts created by the teacher and students, and formal and informal 
interviews of both the teacher and selected students. Formal interviews were recorded with 
audiotape and transcribed, while informal interviews were recorded with hand-written notes. 

The survey research^ comes from coding of student responses to open-ended "project planning 
exercises" from three different CoVis earth science teachers' classes which vary in the amount of 



^At his request, Rory Wagner’s real name is used. All students’ names are pseudonyms. 

wish to thank the entire CoVis staff and all teachers who participated in this survey effort, which was substantial. 
In particular, I want to thank Louis Gomez, who directed the development and piloting of the survey, and Franci 
SteinmuUer, who was the second coder. 
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time devoted to project pedagogy and the degree of responsibility students must take for research 
design. One of the three teachers was Rory Wagner, whose class was also examined in the case 
study. The surveys were administered in the fall and the spring in order to obtain a measure of 
change across the year. Numbers of surveys and data relevant to project-orientation are in Table 1: 



Teacher^ 


Time on 
projects® 


Student 

formulation^ 


Fall 95 
Surveys (n) 


Spring 96 
Surveys (n) 


Project-Based (high 
orientation and time) 


75% 


100% 


50 


20 


Project-Oriented 
(High orientation, 
low time) 


3% 


100% 


72 


94 


Project-Enhanced 
(low orientation, 
moderate time) 


20% 


25% 


84 


68 



Table 1: Groups, project orientation data, and numbers of surveys 



Each set of exercises contained two scenarios, which place the student in the role of director of a 
research project for a scientific problem. There were a total of eight scenarios created, on four 
different topics: beach erosion, global warming, snowstorm prediction, and meteor paths. Each 
scenario came in a short form and a long form, with the latter including example data they could 
use and guiding questions. Students received different scenarios in the Fall and Spring to avoid 
practice effects. Table 2 shows the scenarios for the beach erosion topic: 



■^The designation of teachers as project-based, project-oriented, and project-enhanced is intended to convey varying 
levels of commitment to a project orientation, which in our definition means smdents' participation in research 
design (see smdent formulation column, plus time on projects). Project-based means a very high level, oriented a 
moderate level, and enhanced a piecemeal level. 

^As a percentage of all class time. Based on teacher estimates, observation, and co mm u n ication with teacher. 
^This refers to smdent responsibility for formulation of question (as opposed to assigned by teacher) and analysis 
plan (as opposed to laid out by teacher). Based on teacher estimates, observation, and co mm u ni cation with teacher. 



Beach erosion (short form) 

A small town on the Pacific Ocean is in trouble. They are dependent on tourists who come to their 
town to enjoy their beaches, but the beaches are losing much of their sand. They have hired you to 
act ^ls scientific advisor on beach erosion. Your job is to direct a research program to figure out 
what is causing their beach to erode. Please describe a plan for figuring out the extent and cause of 

erosion at the town's beach, to report to the town council. 

Beach erosion (long form) 

A small town on the Pacific Ocean is in trouble. They are dependent on tourists who come to their 
town to enjoy their beaches, but the beaches are losing much of their sand. They have hired you to 
act as scientific advisor on beach erosion. Your job is to direct a research program to figure out 
what is causing their beach to erode. 

The following is a list of data available to you for your research on beach erosion: 

• Maps showing the coast line and beach size every year for the p^lst 50 years 

• Wave height data for the past 50 years 

• Wind data for the past 50 years 

• Rainfall data for the past ^ years 

• Data on the amount and location of plant life at the beach for the past 50 years 

Please describe a plan for figuring out the extent and cause of erosion at the town's beach, to report 
to the town council. 

To complete your answer, you may find it helpful to use the following questions as a guide: 

Who will you ask to help you? 

What research teams will you form and what will the teams do? 

Which instruments and data will your research teams use? How will they use these instruments 
and data? 

When the research is done, how will you convince others that your conclusions are right? 

Table 2: Project planning scenarios for beach erosion 

One class in depth: Cases of success and frustration 

By his fourth year of conducting a project-oriented earth science class, Rory Wagner has 
developed and refined the way he conducts his class a great deal. He begins the year with a 
quarter-long "whirlwind lecture tour" of the earth sciences, coupled with an introduction to the 
Internet and computer tools his classroom contains (6 computers with the Web, email, spreadsheet 
and word processing software). This is followed by three rounds of quarter long projects which 
his students design and implement with his guidance. In order to help them figure out how to 
conduct projects, Mr. Wagner has developed what can be referred to as an activity structure 
(Lemke, 1990; Polman, 1997; Polman, 1998) for projects. He has broken the overall process of 
open-ended research projects into the following phases for his students, each of which culminates 
in a deliverable artifact or event: ( 1) selection of research group and broad topic, (2) research on 
background information related to the topic, (3) formulation of a research proposal, which usually 
includes a research question and ideas about methods, (4) data collection, (5) data analysis, (6) 
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complete research report preparation, (7) revision of research report based on feedback, and (8) 
presentation of the research to the class. On a November day, students have formed eleven groups 
of two to three and are beginning phase one of their project. 

Bruce, Cheryl, and Sylvia spend most of their time at the computer table just to the right of their 
teacher, Mr. Wagner, at the front of the classroom. Bruce is a tall, quiet and somewhat rumpled 
junior. Cheryl is an outgoing senior who is dramatic in her manner, which is not surprising as she 
is involved in theater. Sylvia is a senior who is graduating early in December. Like many students 
over the three years Rory has been allowing students to choose their own research topics, Bruce, 
Cheryl, and Sylvia express an interest in UFOs and aliens. Rory had been frustrated at all of the 
previous efforts, because the students had been unable to design a research project on UFOs that 
relied on empirical data and argumentation. Despite his misgivings, he decides to let the students 
run with their topic, since he maintains that he cannot predict all the promising avenues students 
might uncover or generate. 

One reason Rory makes a conscious effort to remain open to what may seem at first outlandish 
ideas from students is that it provides motivational benefits to the students when they get to work 
on something they are interested in as well as when they have more ownership of their projects. 
"Kids can come up with some interesting projects ... that you wouldn't think of." An example of 
how a project that appeared problematic at first worked out is one from 1994-95 on "controlling the 
movement of the earth's continental plates" at a location on the San Andreas fault. Trying to stop 
plate movement seemed silly to Rory, and he said he would have rejected the idea in his first year, 
but he decided to let the enthusiastic group try and develop their idea. They ended up doing an 
interesting inquiry in which they determined the size of historical earthquakes at that location, and 
learned about how structures can withstand shearing stresses based on structure and the material 
from which they are made. 

In addition, giving the students real choice on matters that are fundamental to their work, as Lepper 
etal. (1993) mention, means that they are free to make decisions the teacher does not think will be 
best in the long run. To address this problem, Lepper, etal. say expert tutors limit student choices 
to instructionally irrelevant choices and situations "in which the tutor is not certain what would be 
best for the student." In Rory's class, decisions about topics and reseeu’ch questions fall into this 
category. In some cases during this quarter, such as the Woolly Mammoth project and Barb's 
UFOs & Aliens project, Rory's suspicion that the project will go badly is borne out. At the end of 
the Woolly Mammoth project, Rory can see in retrospect that they "got derailed in the beginning" 
from an idea that probably "would have come out better" than the one they chose. Diane, Tom F, 
and Tom M were considering doing something on woolly mammoth extinction or how the woolly 



mammoth and the modem-day elephant occupied similar ecological niches; Rory thought the latter 
idea was much better, but Diane and Tom F's preference for the former led them to pursue it. On 
the other handj in the case of the UFO Sightings project and the Black Holes projects, ideas that 
Rory suspects will be problematic, because they had been tried unsuccessfully by numerous 
students in previous years, result in successful projects. 

As the mixed success of the projects indicates, the problem, as Rory points out, with starting from 
students' interests is that it is "awful hard" in many cases "to transform something you are really 
interested in to something you can do" as scientific research. For effective teaching and learning it 
is not a matter of the teacher simply telling the students what to do; as I have been stressing, Rory 
wants to ensure that students participate in such research design decisions so that they can leam 
about research design. The difficulty and pitfalls of student participation in the whole process of 
research has been recognized by a number of student-scientist collaborative efforts, but even 
though it is often messy from scientists' perspective to have students involved in the whole 
process, it is educationally significant (Pea, Gomez, Edelson, Fishman, Gordin & O'Neill, 1997). 
A balance between student ownership and teacher guidance in potentially promising directions is 
key, allowing both parties to make crucial contributions. As Rory described it, 

sometimes [students] come up with things that are really creative that I would have never 
thought about, which then le^ me to think of other things that might be doable. And 
sometimes— [and] this gets in to the negotiating thing— sometimes they get real close to 
something, or have a neat idea, but it's not doable, so then, how do you turn that into 
something that is doable? Sometimes they do it, sometimes I can do it. 

An example is provided by the way Bruce, Sylvia and Cheryl's project moves from being a project 
about "whether UFOs are alien space ships" (just as Barb's started out and ended up) to a project 
about confirming or supporting natural explanations of UFO sightings. 

Along with the other groups, the UFO Sightings group begin the project by collecting and 
synthesizing background research on the topic, before deciding on a specific research question. In 
their interim report of background research , they mention the so-called "Condon report" (Condon 
& Gillmor, 1968), the only official study of UFO sightings put out by the US government. 
Condon and his colleagues claimed UFO sightings could be explained by meteor showers, rocket 
launches, and other known phenomena 

Two days after he gets the Background Information reports, Rory says to me before class, "I 
should watch out for groups that need support instead of just waiting for it to become a problem. I 
think I'm trying to back off because I don't want to give them a topic and make it my project." 
Given the problematic nature of UFO projects in the past, Cheryl, Bruce and Sylvia are obvious 
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candidates for providing with extra support, and Rory and I are both intrigued with the group's 
description of the Condon report. In our meeting before class, he and I discuss the fact that 
Condon's analysis took an empirical approach based on supportable or refutable claims about 
alternate explanations for UFO sightings. We are both intrigued by how Condon was able to take a 
scientific approach to a problem surrounded by so much hearsay. So during class that, day, Rory 
initiates a discussion with the UFO Sightings group about potential research questions. 



Shortly after completing attendance and answering some procedural questions about the research 
proposal assignment, Rory says, "OK, you guys," to Cheryl, Bruce, and Sylvia, and sits down 
with them. The following interaction takes place: 

Rory: OK, what do you want to do? 

Bruce: We want to show UFOs are alien space ships. 

Rory: [doubtfully] Any ideas on how? 

Bruce: I don't think there's any way to prove it unless they saw the alien in there and they 
waved at them. That's the only evidence there is. 

Rory: Right. That's the problem. 

Cheryl: I don't see why we can't write a report on it if people have written whole books 
on it. [Cheryl sees Rory's project at this point as essentidly the same as an extensive report 
for an English class. As time goes on, she begins to grasp the importance of using 
empirical data to support a claim.] 

Rory: [does not directly address Cheryl's confusion at this time] You know, Joe and I 
were talking about the analysis Condon did that you wrote about in your Background 
Information [report]. It was interesting because Condon claimed to have explained the 
sightings with Imown phenomena. [For your project] you could verify what somebody like 
Condon has done. That's another thing people do in science ..." 



He gives them the example of the cold fusion debate a few years ago, and then points to how this 
could be applied in their project: 

... these guys said they had created cold fusion in the lab. But when other people tried it, 
they couldn't duplicate what they said ... In science, once someone says they've proved 
something, others check it ...The idea is to verify the government's explanations. Say they 
said it was a meteor shower. Y ou could look at the date, where the meteor shower was, 
and when and where people saw the UFO. Does it match the same spot? If the sighting was 
here [points one direction] and the meteor shower there [points another direction] the 
government's explanation could be wrong. 

The students decide to run with the idea. In this example, the students originally present the 
Condon Report as relevant to the history of the UFO debate, and thus something to be cited. 
Through their interaction, Rory and the students create a new meaning for the citation: the seeds of 
a study intended to provide independent confirmation or falsification. 

I will not describe the rest of the UFO Sightings project in much detail. I will note, however, that 
this research formulation succeeds despite the fact that the group is "dysfunctional" in terms of 



attendance (at one point, on only one half of the project days were all three group members present 
in class, and on one sixth of the days only one student was present). In addition, Bruce, Cheryl, 
and Sylvia do not pick up on the technology as quickly as some other groups. In frustration near 
the end of the project when they are trying to assemble their paper in a word processor, Cheryl 
comments "I think I'm gonna turn Amish. I hate computers." Instead of high-tech resources, the 
group almost exclusively uses the library, finding the Condon Report eventually in the 
Northwestern University library, after striking out at the school library, 4 community libraries, and 
the Internet. For their final research report, they choose four UFO sightings from the 1960s 
described in the Condon report, and try to independently confirm or falsify the Condon report's 
explanation. The independent confirmation is based on printed data sources found in library 
searches: a nautical almanac (Casey, etal., 1989) confirmed the position of a planet in the exact 
location where a UFO sighting was reported; NASA launch records (Stanford, 1990) confirmed 
that a scheduled re-entry of satellite Agena into the Earth's atmosphere occurred at the time an 
airplane crew reported a UFO over Mexico, and could have been seen in that location; a daily 
weather book (Thomas, 1979) confirmed that the local conditions matched those associated with 
mirages caused by refraction through warm, dry air. Just as the Condon report claimed. They could 
not confirm or deny the Condon report's assertion that a rocket launch explained a fourth sighting. 

Despite Rory's best efforts to support projects through the activity structure punctuated by 
milestones, and guide students' work, some projects don't turn out well. Although I did not carry 
out detailed observations on the following two project cycles of the 1995-96, the data I have from 
previous years and later in the year suggests that the repetition of the project cycles allows some 
students to improve who have trouble the first time around. For example, one student in 1994-95 
told me that "the second project is going better because we understand Rory's expectations better." 
During the same year, two others did an abysmal project on UFOs during the fall, but made 
significant improvements in research formulation and data analysis during the second round when 
they did a project on geyser eruption patterns in Yellowstone National Park. Later in 1995-96, 
Mark, who worked on the Zodiac project, and Tom F and Tom M, who worked on the Woolly 
Mammoth project, teamed up. The group members chose geysers as their topic and built on some 
of the ideas the group the previous yeeu’ did not finish. These three who had great difficulty in their 
first project gathered data and did an analysis of dormancy patterns in geyser basins that lay 
adjacent to one another. Also later in the year, Patti, Diane, and Marie, who worked on three 
different projects that ran into trouble the first time around, did an interesting project about the 
relationship between the number of tornadoes and the number of deaths caused by tornadoes per 
year. Patti saw the second project as much more manageable than the first one. At the beginning of 
the project in the third quarter, Rory asks the students for feedback on how they think projects 
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could be improved, and one student asks him to "be more specific what [he] want[s]." Rory tries 
to, but for many students, the simple repetition may help more than the way he explains what he 
wants. 

Nonetheless, as it stands, some students just don't have much chance to succeed in Rory's class. 
Two prime examples from this class are Cindy and Barb. Both of them could use more structure 
and guidance than they receive in Rory's class. I have a conversation with Cindy in the middle of 
her second project, that shows she still doesn't understand projects as involving making original 
contributions, and not just synthesizing known facts: 

Joe: What are you up to? 

Cindy: I'm doing volcanoes. 

Joe: What about volcanoes? 

Cindy: Volcanoes in the Pacific Rim, all around the Pacific oceem basically. 

Joe: And where have you gotten to with it? 

Cindy: I've collected all land of information [shows me a pocket folder], and now I have 

to sort it. 

Joe: Do you have any data on it? 

Cindy: What do you mean? 

Joe: Like numbers of eruptions and stuff? 

Cindy: No ... well, sort of. Mostly I've got this information that I have to put together. 

She continues to have procrastination problems, and does not seem to be understanding project 
work any better. She then says she wishes Rory's class were more like the other teachers', where 
they "do labs and take notes and take tests and everything." She also says, "I feel like this class is a 
waste of time. Like I come in here and we're just supposed to work on our projects. I can't focus 
on that during class. I feel like I don't learn anything. I don't know how to do things this way ... 
Some people like it better [this way], but I wish it was more like a normal class." Barb suffers in 
part because she is hesitant to bring questions and problems to her teacher's attention, and he is so 
occupied by questions which more proactive members of the class bring to him that he seldom gets 
to her. 

The question this begs is: what can be done to address the needs of students like Cindy and Barb? 
One possible change is to adjust the level of structure available for students on their second or third 
time around. This could involve offering such students project ideas, for instance from the list of 
promising questions Rory has been accumulating for several years. Even more support could be 
provided if Rory were to recommend such students work on questions for which he knows data 
resources are available, as another CoVis teacher does. Such a strategy could undoubtedly 
introduce or exacerbate other problems in the system, such as questions of fairness given the 
deliberate differences in difficulty of such projects. But to the extent that this strategy could be 
made workable, it would provide a leg up for a student like Cindy, who is uncomfortable with new 



practices. When she was confronted with a computer skills exam early in the year, she had an 
opportunity to get more comfortable by watching others perform successfully; she had much less 
opportunity to become accustomed to new practices without failure in project work. 

These cases demonstrate how students in a class devoted fully to research projects for the last three 
quarters of the year gain the sorts of tools which I argue will serve them well in today's world. 
Nonetheless, the complex role changes for teachers and students makes "project- based" science 
teaching difficult to customize well for all students. In this section, I have described some of the 
successes and difficulties in trying to maintain a balance between the extremes of overly structuring 
student activity and leaving it too open. Since each student and group requires a different level of 
structure and guidance to maintain equilibrium, teaching becomes more difficult. Moving too far in 
one direction or the other compromises both motivation and learning; only by maintaining 
equilibrium can students remain challenged and have maximal opportunities to learn. The examples 
described in this section are intended to show how it is possible, but difficult, to guide student 
work just enough to maintain that equilibrium. 

Several particular directions toward which teachers can steer students became apparent through my 
case study research. In particular, teachers can steer the difficult task of research question 
formulation toward independent confirmation or falsification. This is a potentially important leg up 
for students having trouble formulating research either because their topic is difficult or because 
they lack confidence. To aid in this process, Rory could ask students to look for scientific claims 
during their background information research that they might like to question or see if they can 
independently support during the later phases. In addition, students could look for scientific 
debates like known phenomena as explanations of UFO sightings (and also the question of 
whether a new object identified in space is a black hole, the debate about whether Pluto is a 
"proper" planet, or how Plesiosaurs used their flippers in swimming, all of which Rory's students 
pursued). Such debates may spark student interest and sense of ownership, and demonstrate to 
students that science involves research and argumentation that they as thinking persons can 
participate in. As Beth put it when I asked her what the most interesting part of her project was, "I 
really liked just— it was sort of just like a mystery ... and that I had to like figure it out." 

Obviously, latching onto scientific debates is not a fool-proof recipe for success, as the Dinosaur 
Extinction and the failed UFOs and Aliens projects exemplify. Students need to find debates for 
which they can get and use empirical evidence, and even so, they may still encounter other pitfalls 
along the way. However, the results can be unexpectedly impressive when teacher and students are 
able to maintain a balance between openness and guidance. 
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A broader view: Performance on project planning tasks in different environments 

One problem with case study research such as that done in Rory Wagner's classroom is that it is 
extremely time- and person-intensive. For that reason, it is difficult to do case studies of much 
depth in multiple settings. For the purposes of formative as well as summative evaluation of a 
project-oriented reform effort such as CoVis, however, a broader view is extremely helpful. We 
have conducted a number of multiple choice and Likert-bcised surveys to assess trends in attitudes 
and practices, but we endeavored with the Project Planning Exercises (PPE) to create an instrument 
which could capture richer data on developing student competencies, but still be less time-intensive 
than direct classroom observations. We make the following assumption about the results of these 
open-ended planning tasks: although students who describe a plan of action may not be able to 
carry out all aspects of the task unaided were they given the chance to carry it out"^, students who 
are able to articulate a plan with specific features will be better off than those who do not. For this 
analysis, I will focus on three aspects of the student plans, which relate to the concerns for 
scientific literacy described above: (1) students' unprompted description of specific empirical data 
they would like to use in their inquiry, (2) students' use of specific analysis strategies which we 
see as accepted "epistemic forms" (Collins & Ferguson, in press) of science inquiry, and (3) the 
stated source of solutions to the research questions or problems the students lay out in their plans. 

All student resp>onses were coded for the use of empirical data in their proposed project plan. As 
mentioned in the "Data sources and methods" section, half of the scenarios directly prompted 
students for data, and gave them examples of data; since these "long versions" provided 
scaffolding in the question for the specification of empirical data, and resulted in uniformly high 
use of such data, we did not include them in this analysis. Figure 1 shows the data and chart for 
students pooled according to the project orientation of their teacher: 



^In making this point, we hope to indicate our concurrence with the arguments and research of Lucy Suchman 
(1987), who has shown that abstract plans do not account for all that is involved in situated actions. In order to 
evaluate the situated knowledge of students, we argue that direct observation of student inquiry (as in the case studies) 
is necessary. 



Project-based 

Project-oriented 

Project-enhanced 



Fall 

52 % 

55 % 

54 % 



Spring 

67 % 

63 % 

47 % 




Project- 

based 



Project- 

oriented 



Project- 

enhanced 



Figure 1: Percent of students in 3 teachers' classes specifying data, unprompted 

In the project-enhanced class, where students were not given as much responsibility for 
formulating research and analysis design, the number of students specifying empirical data to 
collect for the project actually decreased. On the other hand, when students spent a small amount of 
time on research design in the project-oriented class, but had a teacher who emphasized it, more 
specified empirical data to use in the inquiry. 

We also coded student responses for the sorts of common analysis strategies they specified for 
their planned inquiry. Table 3 shows the categories which we coded as binary items (simply 
present or not in all responses), which were then grouped into broader categories that reflect our 
valuation of increasing sophistication and specificity to natural science. In addition to the presence 
or absence of each strategy, we determined the highest level broad category of strategy found in 
each response, and used that numeric data to compare the pools of students taught by the three 
teachers. 



Binary code 


Explanation 


Broad Category 


Discover 


They find answer by looking or "researching" or 
"studying," but there is no indication of one of the 
following ways of looking 


0. Discover 


Find patterns 


Find patterns of some sort in some data, but ill- 
Sfjecified 


1. Inconclusive Strategies 


Invoice strategy 


They simply name a strategy but they do not make 
it clear in any way that they know what that 
strategy could mean in this case. The use of the 
term may be token. For example "I would make up 
some kind of hypothesis of where the meteor is 
going to hit." In this case, they don't indicate what 
the hypothesis would be or how or even that they 
would test it. 


1. Inconclusive Strategies 


Graph / chart / 
summarize 


Graph or chart some variables, or perform 
summary calculations such as averaging on a 
variable 


1. Inconclusive Strategies 



Copy 


Copy another solution and if necessary modily 
what's been done elsewhere 


2. General Strategies 


Extrapolate 
from current 
conditions 


Determine what the relevant conditions are at 
present and assume they will continue in a 
predictable way. An example is assuming the 
weather to the west of a location will continue as is 
and affect that location. 


2. General Strategies 


Match current 
case with 
historical cases 


Try to match the configuration of some 
conditions/variables with identical conditions in the 
past. 


2. General Strategies 


Predict and test 


They specify a prediction about what will happen, 
and test it in situ or in historical data. 


3. Experimental Strategies 


Verify 

hypothesized 

cause 


They will try to see if a cause they predicted is 
borne out (that cause must be specified). 


3. Experimental Strategies 


Laboratory 

experiment 


They describe a lab experiment which will 
contribute to their analysis. 


3. Experimental Strategies 


Model testing 


They describe a physicd or analytical model they 
will use to perform tests. 


3. Experimental Strategies 


Covariation or 
co-occurence of 
variables 


They look for covarying quantitative variables— 
e.g., car exhaust increase corresponds with 
temperature increase — or co-occurence of variables 


4. Numerical Strategies 


Formula 


They create a formula or mathematical model that 
expresses the relationship between variables. 


4. Numerical Strategies 



Table 3: Analysis strategies coding scheme 



Once again, results from students in the project-enhanced class, where little work on design of 
research and analysis was done, were discouraging. The percentage of students in the Project- 
Enhanced class who stated that they would simply "research" in a completely unspecified way to 
find the answer increased from 27% in the Fall to 44% in the Spring. In the project-based class, 
there was a large increase in broad category 1 responses ("General Strategies") — from 22% in the 
Fall to 31% in the Spring. This is probably attributable to the emphasis that the teacher put on 
graphing and finding patterns in data. In the project-oriented class, there was a moderate increase 
in General Strategies (from 20% to 25%), and a large increase in numerical strategies (from 4% to 
12%). The project-oriented teacher may have worked on numeric strategies outside of projects as 
well, perhaps in lab tasks. 



Another part of our coding involved looking at the source of the given or eventual analytic solution 
or conclusions to the planned inquiry. We were particularly concerned with whether students 
clearly saw their own contribution as vital to eventual conclusions, rather than relying completely 
on experts or printed sources. We believe this might indicate their own perception of themselves as 
at least "legitimate peripheral participants" in scientific inquiry. Table 4 shows the coding scheme: 



Number I Code 



Explanation 



0 


Unclear 


It is unclear what the source of the solution would be. There 
is not enough information provided 


1 


Expert or Printed T ext 


They leave solving the problem entirely to the experts, which 
does not include them, or they expect to find the solution 
written out in a book, magazine, or journal. 


2 


Student's unspecified 
previous knowledge 


They assert a solution they know somehow already, 
precluding the need for further investigation. 


3 


Own original analysis 


They plan on using their own original analysis to formulate a 
solution or conclusion 


4 


Combo (own + expert) 


Combination of their own analysis with expert input or 
printed text 


5 


Consensus 


There is an explicit process of trying to reach consensus in a 
group with possibly conflicting viewpoints to achieve greater 
veracity 



Table 4: Solution Source coding scheme 



As in the previous two examples, the students in the project-enhanced did not progress in the 
direction we would have hoped. Whereas fewer students in the other classes expressed a reliance 
or experts or print text sources (category 1) for answers in the Spring of the year (6% in the Spring 
vs. 9% in the Fall, for both groups), more students in the project-enhanced class said they would 
rely on experts or print sources in the Spring ( 12%) than did in the Fall (9%). On the other hand, 
the percentage of students who fell into categories 3-5 in the project-oriented and project-based 
classes increased— moderately in the project-based class from 55% to 60%, and strongly in the 
project-oriented class from 63% to 72%. 





Reliance on ot 


lers (code 1) 


Reliance on own analysis (codes 3-5) 


Fall 


Spring 


Fall 


Spring 


FYoject-based classes 


9% 


6% 


55% 


60% 


Project-oriented classes 


9% 


6% 


63% 


72% 


Project-enhanced classes 


9% 


13% 


59% 


60% 



Table 2: Sources of solutions in students' responses 



Thus, it appears students in the project-based and project-oriented classes felt more empowered to 
play a role in solving scientific problems by their experiences, whereas students in the project- 
enhanced class felt less emjxjwered. 

Overall, these results indicate that involving students in research projects where they are challenged 
to formulate the research design and the analysis strategies for coming to conclusions appear to be 
essential to developing some of the competencies and attitudes I have emphasized. The relative 
strength of results from the project-oriented class, where very little class time was spent on 
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projects, would seem to indicate that such efforts need not occupy a huge portion of the 
curriculum. But there are two caveats: first, we do not know how much out of class work students 
were asked to do in this class, and second, we do not know whether the students in the project- 
based class may have developed more situated competence in the actual carrying out of their plans. 
Both of these questions could be addressed by further case study research. 

Conclusion 



In this paper I argued that certain aspects of citizenship in democratic societies today require 
something other than a knowledge of science as an accumulation of facts. In particular, I focused 
on the prevalence and importance of decisions related to science and science research studies, 
ranging from individual decisions made by citizens with their doctors to environmental policy. In 
this I side with Deborah Meier, the principal of Central Park East Secondary School, and author of 
The Power of their Ideas (Meier, 1995). She recommends that we foster "the capacity to hazard an 
opinion on matters of science that may pertain to political and moral priorities, and a healthy and 
knowing skepticism toward the misuse of scientific authority" (p. 168). 



In addition, I have presented evidence that reaching these goals may be well-served by teaching 
children to be "little scientists" to a certain degree— by conducting research projects. Although 
more research is needed on tradeoffs involving the implications of relative emphasis and amount of 
time spent on projects, involving students in projects that challenge them to participate in the design 
of research and analysis strategies have a demonstrable benefit. A final example from Rory 
Wagner's project-based class illustrates the development of a capacity for engaging in political 
matters pertaining to science. In her firetj roiect of the year, Beth was helped by Mr. Wagner to 
see that claims about the phenomenon of plesiosaur swimming motion need not be accepted as 
simple "fact" or "fiction," and together they figured out some strategies for independently 
confirming or falsifying the claims by assembling independent data. In her ne^ p roject, Beth chose 
underground nuclear testing as her topic, and soon encountered claims from environmental 
organizations and the French government that such testing causes geologic damage. After doing 
further research on underground nuclear testing, she suggested, with no prompting from her 
teacher, that the environmental organizations were making catastrophic claims without data to 
support their conclusions. She had, in an important sense, appropriated some sense of the need for 
and nature of adequate evidence to support scientific claims. 
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